Exploiting Audio-visual Correlation in Coding of Talking Head Sequences

TALKING HEAD SEQUENCES Ram R. Rao Georgia Institute of Technology Atlanta, GA 30332 [email protected] Tsuhan Chen AT&T Bell Laboratories Holmdel, NJ 07733 [email protected] ABSTRACT In this paper, we present a novel means for predicting the shape of a person's mouth from the corresponding speech signal and explore applications of this prediction to video coding. One possible application is cross-modal predictive coding. In the cross-modal predictive coding system described in this paper, a model-based video coder compares measured visual parameters with predicted visual parameters, and sends the di erence between the two to the receiver. Since the decoder also receives the acoustic data, it can form the prediction and then reconstruct the original parameters by adding the transmitted error signal.